04. Fully Connected to 1x1 Convolution

1x1 Convolutions

Let us recap a few things before we move on to implementing a 1x1 convolution in TensorFlow.

In a previous lesson we covered how, in TensorFlow, the output shape of a convolutional layer is a 4D tensor. However, when we wish to feed the output of a convolutional layer into a fully connected layer, we flatten it into a 2D tensor. This results in the loss of spatial information, because no information about the location of the pixels is preserved.

We can avoid that by using 1x1 convolutions.

Quiz: 1x1 Convolutions

A 1x1 convolution is essentially convolving with a set of filters of dimensions:

• 1x1xfilter_size (HxWxD),
• stride = 1, and
• zero (same) padding.

Consider the image above. A 1x1 convolution is applied to the output of a regular convolution. Let's assume that the regular convolution output is of dimensions 7x7x20 (HxWxD). These dimensions become the input to the 1x1 convolution with a filter depth of 5.

What would be the dimension of the output of this convolution, the layer shown in red above? (Hint: Make sure to check out the formula at the end of this section!)

As you observed from the quiz above, 1x1 convolution helped in reducing the dimensionality of the layer. A fully-connected layer of the same size would result in the same number of features. However, replacement of fully-connected layers with convolutional layers presents an added advantage that during inference (testing your model), you can feed images of any size into your trained network.

Next, you will learn to rewrite a fully-connected (dense) layer to a convolutional layer in TensorFlow!